Electronic apparatus providing content-based care service and controlling method thereof

ABSTRACT

An electronic apparatus is provided. The electronic apparatus may include a display; a camera; a memory in which reference information associated with a content that provides information is stored; and a processor may be configured to obtain an image photographing a user through the camera while the content is provided through the display; obtain context information associated with the user based on the image; identify an execution status of the user associated with the information provided by the content by comparing the obtained context information and the reference information; and control an output state of the content based on the execution status of the user.

CROSS REFERENCES TO RELATED APPLICATIONS

The present application is a bypass continuation of InternationalApplication No. PCT/KR2021/015211, filed on Oct. 27 2021, in the KoreanIntellectual Property Office, which claims priority from Korean PatentApplication No. 10-2020-0180222, filed on Dec. 21, 2020, in the KoreanIntellectual Property Office, the disclosures of which are incorporatedherein in their entireties.

FIELD

This disclosure relates to an electronic apparatus that provides a careservice related to a content for providing information and a controllingmethod thereof.

BACKGROUND

Recently, as platforms based on User Generated Content (UGC) such asYouTube have become popular, content that can satisfy the needs ofvarious users are being produced and distributed. Accordingly, robotsthat can provide such content to users are being actively developed.However, existing robots only perform or provide unidirectional contentto users, and have difficulties in providing satisfactory servicesbecause they do not reflect changes in the characteristics andcircumstances of the users that receive the content. Therefore, therehas been a continuous demand for a method of providing user-friendlycare services through interaction with users in the process of providingcontent.

SUMMARY

The present disclosure relates to an electronic apparatus that providesa user-friendly care service through an interaction with the user in theprocess of providing content and a controlling method thereof.

An electronic apparatus according to an embodiment may include adisplay; a camera; a memory in which reference information associatedwith a content that provides information is stored; and a processor maybe configured to obtain an image photographing a user through the camerawhile the content is provided through the display; obtain contextinformation associated with the user based on the image; identify anexecution status of the user associated with the information provided bythe content by comparing the obtained context information and thereference information; and control an output state of the content basedon the execution status of the user.

The content may include sub information corresponding to each of aplurality of steps, and the processor may be configured to, while subinformation corresponding to one step from among the plurality steps isprovided through the display, identify the user's execution statusregarding the one step by comparing the obtained user's contextinformation and reference information corresponding to the subinformation, and control an output state of the content related to theone step based on the user's execution status.

The memory may store information regarding an output type of content foreach execution level of the user, and the processor may be configured toidentify an execution level corresponding to the user's executionstatus, identify information regarding an output type of a contentcorresponding to the identified execution level among information storedin the memory, and control an output state of the content based on theidentified output type.

The output type of the content may include at least one of still imageoutput, repeated output of sub information corresponding to a specificstep, output speed adjustment, or enlarged output of the content.

The apparatus may further include a driving unit, and the processor maybe configured to obtain the user's posture information based on theimage obtained while a content that provides health informationincluding an exercise posture is provided through the display, identifythe user's execution status regarding the exercise posture by comparingthe obtained posture information and posture information included in thereference information, and control the driving unit to provide adifferent feedback for guiding the user's posture based on the user'sexecution status.

The electronic apparatus may be implemented as a robot that provides ahealthcare service, and the robot may perform an operation of correctingthe user's posture or provides guide information for guiding the user'sposture so that the user executes an exercise posture provided by thecontent based on the user's execution status.

The processor may be configured to identify the user's execution levelbased on at least one of a degree of correspondence between the obtainedposture information and posture information included in the referenceinformation, a frequency of discrepancy between the obtained postureinformation and posture information included in the referenceinformation, or a complexity of the exercise posture, and provide afeedback corresponding to the identified execution level.

The processor may be configured to obtain the user's status informationbased on the image obtained while a content that provides taskinformation including a plurality of steps is provided through thedisplay, identify the user's execution status regarding each of theplurality of steps by comparing the obtained status information andstatus information included in the reference information, and controlthe display to provide guide information corresponding to a steprequiring a feedback from among the plurality of steps based on theuser's execution status.

The processor may be configured to obtain the user's first statusinformation by inputting the image to a first neural network model,obtain the user's second status information by inputting the user'svoice to a second neural network model, and control the display toprovide guide information corresponding to a step requiring a feedbackfrom among the plurality of steps based on the identified executionstatus, the first status information and the second status information.

The apparatus may further include a speaker, and the processor may beconfigured to, based on an image of the user not being obtained whileinformation regarding one step from among the plurality of steps isprovided, control the speaker to output guide information related to thetask information.

A controlling method of an electronic apparatus according to anembodiment may include obtaining an image photographing a user through acamera while a content that provides information is displayed; obtainingcontext information associated with the user based on the image;identifying an execution status of the user associated with theinformation provided by the content by comparing the obtained contextinformation and the reference information associated with the content;and controlling an output state of the content based on the executionstatus of the user.

The content may include sub information corresponding to each of aplurality of steps, the identifying the user's execution status mayinclude, while sub information corresponding to one step from among theplurality steps is provided through the display, identifying the user'sexecution status regarding the one step by comparing the obtained user'scontext information and reference information corresponding to the subinformation, and the controlling an output state of the content mayinclude controlling an output state of the content related to the onestep based on the user's execution status.

The identifying the user's execution status may include identifying anexecution level corresponding to the user's execution status, and thecontrolling an output state of the content may include identifyinginformation regarding an output type of content corresponding to theidentified execution level and controlling an output state of thecontent based on the identified output type.

The output type of the content may include at least one of still imageoutput, repeated output of sub information corresponding to a specificstep, output speed adjustment, or enlarged output of the content.

The obtaining context information related to the user may includeobtaining the user's posture information based on the image obtainedwhile a content that provides health information including an exerciseposture is provided through the display, and the identifying the user'sexecution status may include identifying the user's execution statusregarding the exercise posture by comparing the obtained postureinformation and posture information included in the referenceinformation and providing a different feedback for guiding the user'sposture based on the user's execution status.

According to various embodiments, a satisfactory care service may beprovided to users who have different levels of understanding ordifferent execution statuses regarding information provided throughcontent.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an exemplary view illustrating a content providing methodregarding a user of an electronic apparatus;

FIG. 2 is a block diagram illustrating an exemplary configuration of anelectronic apparatus according to an embodiment;

FIG. 3 is a block diagram illustrating an exemplary functionalconfiguration of an electronic apparatus according to an embodiment;

FIG. 4 is an exemplary view illustrating a step included in a contentand reference information for each step;

FIG. 5 is an exemplary method of identifying a user's execution statusbased on posture information according to an embodiment;

FIGS. 6A and 6B are exemplary views illustrating identifying anexecution level corresponding to a user's execution status according toan embodiment;

FIGS. 7A and 7B are views illustrating a posture correction operationand a mimic training operation of an electronic apparatus according toan embodiment;

FIGS. 8A to 8C are views illustrating controlling an output state of acontent by recognizing a user's facial expression by an electronicapparatus according to an embodiment;

FIGS. 9A to 9C are views illustrating controlling an output state of acontent by recognizing a user's voice by an electronic apparatusaccording to an embodiment;

FIG. 10 is a functional block diagram illustrating the functionalconfiguration of an electronic apparatus according to an embodiment; and

FIG. 11 is a flowchart illustrating a controlling method according to anembodiment.

DETAILED DESCRIPTION

Hereinafter, the present disclosure will be described with reference tothe accompanying drawings.

The terms used in the example embodiments of the disclosure are generalterms which are widely used now and selected considering the functionsof the disclosure. However, the terms may vary depending on theintention of a person skilled in the art, a precedent, or the advent ofnew technology. In addition, in a specified case, the term may bearbitrarily selected. In this case, the meaning of the term will beexplained in the corresponding description. Therefore, terms used in thedisclosure may be defined based on a meaning of the terms and contentsdescribed in the disclosure, not simply based on names of the terms.

As used herein, the expression “have”, “may have”, “include”, or “mayinclude” refers to the existence of a corresponding feature (e.g.,numeral, function, operation, or constituent element such as component),and does not exclude one or more additional features.

The expression of “at least one of A and/or B” is to be understood asindicating any one of “A” or “B” or “A and B”.

The expression “a first”, “a second”, “the first”, or “the second” usedin various example embodiments of the disclosure may modify variouscomponents regardless of their order and/or the importance but does notlimit the corresponding components.

When it is mentioned that any component (for example, a first component)is (operatively or communicatively) coupled to or is connected toanother component (for example, a second component), it is to beunderstood that any component is directly coupled to another componentor may be coupled to another component through the other component (forexample, a third component). On the other hand, when it is mentionedthat any component (for example, a first component) is “directlycoupled” or “directly connected” to another component (for example, asecond component), it is to be understood that the other component (forexample, a third component) is not present between any component andanother component.

A singular expression includes a plural expression as long as they areclearly distinguished in the context. In the application, it should beunderstood that the terms such as “comprising”, “including” are intendedto express that features, numbers, steps, operations, constituentelements, part, or combinations thereof described in the specificationare present and do not exclude existence or additions of one or moreother features, numbers, steps, operations, constituent elements, part,or combinations thereof.

In the disclosure, the term “module” or “unit” performs at least onefunction or operation, and may be embodied as hardware, software, or acombination thereof. A plurality of “modules” or a plurality of “units”may be integrated into at least one module to form at least oneprocessor (not shown), except a “module” or “unit” which needs beembodied as particular hardware.

In the disclosure, the term “user” may refer to a person that uses anelectronic apparatus. Hereinafter, an embodiment of the presentdisclosure will be described in greater detail with reference to theaccompanying drawings.

FIG. 1 is a view provided to explain a content providing methodregarding a user of an electronic apparatus.

Referring to FIG. 1 , an electronic apparatus 10 may provide a content20 that provides information to a user 30. Here, the content 20 may be ahealth-related content that describes a squat action.

The user 30 may follow a specific action based on information providedthrough the content 20 provided by the electronic apparatus 10. Theaction of the user 30 according to an embodiment may be an action offollowing a squat action included a health-related content.

The action of the user 30 according to another embodiment may be anaction of following an origami action included in a kids-related contentor an action of following an action of preparing ingredients included ina cooking-related content.

Meanwhile, the user 30 may only accept and understand informationprovided by the content 20, but may not following an action included inthe content 20. The term ‘execution status’ used in the specificationrefers to a status related to information regarding whether the userunderstands information included in the content provided by theelectronic apparatus or information regarding whether the useraccurately follows a specific action based on information included inthe content.

A plurality of users receiving the content 20 through the electronicapparatus 10 may have different abilities to perform the above-describedaction according to individual body characteristics and understanding offields related to information provided through the content.

Specifically, a user with inflexible lower body may have difficulty infollowing a squat action, a color-weakened user may not be able tofollow an origami action quickly, and a user who is not skilled incutting may have difficulty in accurately following an action ofpreparing ingredients.

Therefore, in executing an action included in the content, each of theplurality of users who are provided with the content has differentexecution status. Nevertheless, existing electronic apparatuses have aproblem in that they cannot provide a satisfactory service to usersbecause they do not consider the execution status of the users.

Accordingly, hereinafter, various embodiments in which an electronicapparatus may provide a care service considering a user's executionstatus through an interaction with the user will be described in greaterdetail. In the present specification, the expression that an electronicapparatus ‘provides an interaction’ to a user and the expression that anelectronic apparatus ‘provides a feedback’ are used interchangeablybased on the premise that the two expressions have the same meaning.

FIG. 2 is a block diagram provided to explain configuration of anelectronic apparatus according to an embodiment.

Referring to FIG. 2 , an electronic apparatus 100 according to anembodiment may include a display 110, a camera 120, a memory 130, and aprocessor 140.

The display 110 may be implemented as various types of displays such asLiquid Crystal Display (LCD), Organic Light Emitting Diodes (OLED)display, Quantum dot light-emitting diodes (QLED) display, PlasmaDisplay Panel (PDP), etc. The display 110 may include a driving circuit,a backlight unit, and the like, which may be implemented in the formsuch as an a-si thin film transistor (TFT), low temperature poly silicon(LTPS) TFT, or an organic TFT (OTFT). Meanwhile, the display 110 may beimplemented as a flexible display, a 3D display and the like.

The camera 120 may obtain an image by capturing an area within a Fieldof View (FoV) of the camera.

The camera 120 may include a lens that focuses visible light or a signalreflected by an object, for example, a user, to an image sensor and animage sensor capable of detecting the visible light or the signal. Here,the image sensor may include a 2D pixel array that is divided into aplurality of pixels.

In addition, the camera 120 according to an embodiment may beimplemented as a depth camera operating in a Time-Of-Flight (ToF)method.

The memory 130 may store data necessary for various embodiments of thepresent disclosure. The memory 130 may be implemented as a memoryembedded in the electronic apparatus 100, or implemented in a memoryform capable of being detachable from the electronic apparatus 100,based on a data storage purpose. For example, data for driving theelectronic apparatus 100 may be stored in the memory embedded in theelectronic apparatus 100, and data for an extension function of theelectronic apparatus 100 may be stored in the memory capable of beingdetached from the electronic apparatus 100. Meanwhile, when implementedas the memory embedded in the electronic apparatus 100, the memory 130may be implemented as at least one of a volatile memory (e.g., dynamicrandom access memory (DRAM), static RAM (SRAM), or synchronous dynamicRAM (SDRAM)), or a non-volatile memory (e.g., one time programmable readonly memory (OTPROM), programmable ROM (PROM), erasable and programmableROM (EPROM), electrically erasable and programmable ROM (EEPROM), maskROM, flash ROM, flash memory (e.g., NAND flash, or NOR flash), harddrive, or solid state drive (SSD)). When implemented as the memorycapable of being detached from the electronic apparatus 100, the memory130 may be implemented in the form of a memory card (e.g., compact flash(CF), secure digital (SD), micro secure digital (Micro-SD), mini securedigital (Mini-SD), extreme digital (xD), or multi-media card (MMC)), oran external memory (e.g., USB memory) which may be connected to auniversal serial bus (USB) port.

The memory 130 according to an embodiment may store referenceinformation related to a content that provides information. Here, thereference information may be information corresponding to informationincluded in the content that provides information.

The processor 140 controls the overall operations of the electronicapparatus 100. Specifically, the processor 140 may be connected to eachcomponent of the electronic apparatus 100 and control the overalloperations of the electronic apparatus 100. For example, the processor140 may be connected to the display 110, the camera 120 and the memory130 and control the operations of the electronic apparatus 100.

According to an embodiment, the processor 140 may be referred to variousnames such as digital signal processor (DSP), microprocessor, centralprocessing unit (CPU), micro controller unit (MCU), micro processingunit (MPU), neural processing unit (NPU), controller, applicationprocessor (AP), but it will be referred to as the processor 140 in thepresent specification.

The processor 140 may be implemented as System on Chip (SoC) or largescale integration (LSI), and it may also be implemented in the form ofField Programmable gate array (FPGA). In addition, the processor 140 mayinclude a volatile memory such as SRAM.

The processor 140 according to an embodiment may obtain an imagecapturing a user through the camera 120 while a content is providedthrough the display 110.

The content according to an embodiment may be a content related tohealth, kids or cooking, but is not limited thereto. The processor 140according to an embodiment may control the camera 120 to obtain an imageincluding at least one of the user's front, side or back.

In addition, the processor 140 according to an embodiment may obtaincontext information related to the user based on the obtained image. Thecontext information according to an embodiment may include informationregarding the user's execution status corresponding to informationprovided through the content among information included in the obtainedimage.

Specifically, the context information according to an embodiment mayinclude at least one of the posture information of the user who isreceiving the content, the user's facial expression information or theuser's voice information.

The processor 140 according to an embodiment may identify the user'sexecution status regarding information provided by the content bycomparing the obtained context information and reference informationstored in the memory 130.

In addition, the processor 140 according to an embodiment may controlthe output state of the content based on the identified user's executionstatus. Specifically, the processor 140 may change and provide theoutput type of the content that is being provided. The output type ofcontent according to an embodiment may include at least one of stillimage output, repeated output of specific information included in thecontent, output speed adjustment, or enlarged output of the content.

The content according to an embodiment may consist of a plurality ofsteps including sub information. Specifically, when the contentaccording to an embodiment is a health-related content, the content mayconsist of a plurality of steps including sub information correspondingto each action of exercise.

The processor 140 according to an embodiment may identify the user'sexecution status regarding one step by comparing the obtained user'scontext information and reference information corresponding to subinformation while the sub information corresponding to one step fromamong a plurality of steps is provided through the display.

Specifically, when the content according to an embodiment is ahealth-related content, the processor 140 according to an embodiment mayidentify the user's execution status regarding a step corresponding toeach action of exercise by comparing context information including theuser's exercise posture information and reference informationcorresponding to sub information corresponding to each action.

The processor 140 according to an embodiment may control the outputstate of the content related to one step based on the identified user'sexecution status.

The processor 140 according to an embodiment may identify an executionlevel corresponding to the user's execution status. Here, the executionlevel according to an embodiment may be a level regarding how good theuser's execution status is.

The processor 140 according to an embodiment may control the outputstate of the content based on information regarding the output type ofcontent for each execution level of the user stored in the memory andthe user's execution level.

When the content according to an embodiment consists of a plurality ofsteps including sub information, the processor 140 according to anembodiment may control the output state of the content so that subinformation corresponding to a specific step included in the content isoutput repeatedly.

When the content according to an embodiment is a health-related content,the content may provide health information including an exerciseposture. The processor 140 according to an embodiment may obtain theuser's posture information based on the image obtained while the contentis provided through the display 110.

In addition, the processor 140 according to an embodiment may identifythe user's execution status regarding the exercise posture by comparingthe obtained posture information and posture information included inreference information. The processor 140 may control a driving unitprovided in the electronic apparatus so as to provide differentfeedbacks for guiding the user' posture based on the identifiedexecution status. Specifically, the processor 140 according to anembodiment may control the driving unit to provide different feedbacksincluding at least one of driving of the electronic apparatus 100 ordriving a mechanical part provided in the electronic apparatus 100.

Here, the processor 140 according to an embodiment may identify theuser's execution level based on at least one of a degree ofcorrespondence between the obtained posture information and postureinformation included in the reference information, a frequency ofdiscrepancy between the obtained posture information and postureinformation included in the reference information, or complexity of theexercise posture.

The electronic apparatus 100 according to an embodiment may beimplemented as a robot that provides a healthcare service, and the robotaccording to an embodiment may provide an operation of correcting theuser's posture as a feedback so that the user executes an exerciseposture provided by the content based on the user's execution status.

In addition, the robot may provide an operation of providing guideinformation for guiding the user's posture as a feedback. This will bedescribed in detail with reference to FIGS. 7A and 7B.

The processor 140 according to an embodiment may obtain the user'sstatus information based on the image obtained while the content thatprovides task information is provided through the display 110. Here, thetask information according to an embodiment may be information includinga plurality of steps which are combined based on priorities betweensteps.

Specifically, among step 1, step 2 and step 3 included in the taskinformation according to an embodiment, step 2 may be a step initiatedon the premise of completion of step 1, and step 3 may be a stepinitiated on the premise of completion of step 1 and step 2.

The processor 140 according to an embodiment may identify the user'sexecution status for each of the plurality of steps by comparing theobtained status information and status information included in thereference information, and control the display 110 to provide guideinformation corresponding to a step requiring a feedback from among theplurality of steps based on the user's execution status.

The processor 140 according to an embodiment may obtain the user's firststatus information by inputting the obtained image to a first neuralnetwork model, and obtain the user's second status information byinputting the user's voice to a second neural network model.

The electronic apparatus 100 according to an embodiment may furtherinclude a speaker, and when the user's image is not obtained whileinformation regarding one of the plurality of steps is provided, theprocessor m140 according to an embodiment may control the speaker tooutput guide information related to task information.

Specifically, when the user's image is not obtained, the processor 140according to an embodiment may identify that the user stops watching thecontent, and control the speaker to output a guide voice informing thatthe content has been stopped. Meanwhile, the processor 140 may controlthe speaker to output a guide voice requesting the user to concentrateon the content.

FIG. 3 is a block diagram provided to explain functional configurationof an electronic apparatus according to an embodiment.

Each block illustrated in FIG. 3 may correspond to a plurality ofmodules related to functional configuration of the electronic apparatus100. Here, the plurality of modules according to an embodiment may be asoftware module stored in a memory of the electronic apparatus 100 or ahardware module implemented as a circuit in the electronic apparatus100. Alternatively, each of the plurality of modules may be implementedas a combination of software and hardware.

In the present disclosure, the functions of the electronic apparatus 100will be described based on the assumption that functions performed bythe plurality of modules are performed by the processor 140 which is onecomponent of the electronic apparatus 100.

Referring to FIG. 3 , the electronic apparatus 100 according to anembodiment may provide a user 300 with a service through a task model 20and an interaction model 30. The task model 20 according to anembodiment may include a content extractor 21 and a context analyzer 22,and the interaction model 30 according to an embodiment may include aposture analyzer 31, an execution evaluator 32, a face/voice recognizer33, and an intention analyzer 34.

A content 200 according to an embodiment may consist of a plurality ofsteps including sub information.

The content extractor 21 according to an embodiment may extract validinformation from a video or text-based content 200. When the content 200according to an embodiment is an image-based content, the contentextractor 21 performs the function of extracting a content areaaccording to a predetermined extraction rule from the image providedthrough the display 110.

The context analyzer 22 according to an embodiment may obtain a contextbased on information included in the content 200 that is providedthrough the extracted content area. Specifically, the context analyzer22 may obtain a context based on a change in image, voice, subtitle,caption or the like included in the content 200.

The context analyzer 22 according to an embodiment may identify aplurality of steps consisting the content 200 based on the obtainedcontext. When the content 200 according to an embodiment is ahealth-related content informing a squat action, the context analyzer 22may identify a plurality of steps corresponding to ‘taking a startposture’, ‘sitting down’, ‘standing up’ and ‘standing upright’ based oninformation included in the content.

The task model 20 according to an embodiment may identify ideal forms ofthe identified plurality of steps based on reference informationcorresponding to the plurality of steps identified by the contextanalyzer 22.

The reference information according to an embodiment may be informationgenerated based on the obtained context, but may be information storedin the memory 130 that is included in the electronic apparatus 100 basedon specialized data. The reference information according to anembodiment may include main points and difficulties for steps as shownin Table 1 below.

TABLE 1 Step Main point Complexity 1) Take a Leg width Easy startposture 2) Sit down Angle between Medium thigh and shin Curvature ofwaist 3) Stand up Balance of left and Hard right sides of body Directionof gaze 4) Stand upright Match rate with Easy start posture

With respect to a step corresponding to ‘take a start posture’, the taskmodel 20 according to an embodiment may identify a case in which thedifference between the leg width and the shoulder width is equal to orgreater than a threshold value as an ideal form regarding thecorresponding step. The interaction model 30 according to an embodimentmay include a posture analyzer 31, an execution evaluator 32, aface/voice recognizer 33, and an intention analyzer 34. In addition, theinteraction model 30 according to an embodiment may exchange informationwith the task model 20.

The posture analyzer 31 according to an embodiment analyzes a posture ofthe user 300. In FIG. 3 , the term ‘a posture analyzer’ is used, butwhen information included in the content 200 is information related to a‘result’ rather than a posture, the posture analyzer 31 may analyze aresult made by the user 300 from the image obtained through the camera120.

The posture analyzer 31 may obtain context information related to theuser by analyzing a posture. Here, the context information related tothe user may be information corresponding to the user's posture.

In addition, the posture analyzer 31 according to an embodiment mayanalyze a posture of the user 300 through Skeleton analysis which willbe described in greater detail with reference to FIG. 5 .

The execution evaluator 32 according to an embodiment may identify theuser's execution status by comparing the user's posture informationobtained through analysis in which sub information corresponding to oneof a plurality of steps is provided and reference informationcorresponding to the sub information. When the content 200 according toan embodiment is a health-related content, the execution analyzer 32 mayidentify the user's execution status regarding the corresponding step bycomparing the ideal form of ‘take a start posture” which is one stepincluded in the content and context information related to the user.

The face/voice recognizer 33 according to an embodiment may recognizethe user's face and voice. Specifically, the face/voice recognizer 33according to an embodiment may obtain the user's facial informationbased on an image capturing the user's face or obtain an utteranceintention included in a voice by recognizing the user's voice.

The intention analyzer 34 according to an embodiment may identify alevel of understanding of the user 300 regarding information provided bythe content based on the obtained facial expression information or theutterance intention information.

When it is difficult to understand information provided by the content,the user 300 may make a specific facial expression or utter specificcontent. Specifically, the specific facial expression may be a frowningexpression, and the specific content according to an embodiment may becontent implying that information provided through the content 200 isdifficult to understand.

Specifically, when the user's frowning expression is included in theimage obtained while sub information corresponding to a specific step isprovided, the intention analyzer 34 according to an embodiment mayidentify that the user's understanding for the corresponding step is nothigh.

Meanwhile, the intention analyzer 34 may obtain utterance intentioninformation included in a voice by recognizing the user's voice, andidentify the user's level of understanding based on the obtainedutterance intention information.

The interaction model 30 according to an embodiment may determine aninteraction to be provided to the user 300 based on information exchangewith the task model 20.

Specifically, the interaction model 30 may provide different types ofinteractions based on the user's execution status or the user's level ofunderstanding regarding information provided through the content 200.

The interaction model 30 according to an embodiment may determine aninteraction to be provided to the user 300 based on not only the user'sexecution status and level of understanding but also factors unrelatedto the user's characteristics such as complexity of individual stepsconstituting a content and specifications of electronic apparatuses.

The electronic apparatus 100 according to an embodiment based onfunctions performed by the above-described plurality of modules mayprovide an interaction appropriate for the user 300 in consideration ofthe user's execution status and understanding, improving the user'ssatisfaction for services.

FIG. 4 is a view provided to explain a step included in a content andreference information for each step.

Referring to FIG. 4 , the content 200 may be provided on the display 110provided in the electronic apparatus 100 according to an embodiment.Here, the content 200 according to an embodiment may be a health-relatedcontent that informs a squat action.

The processor 140 according to an embodiment may identify a plurality ofsteps included in a content 401. Specifically, the processor 140 mayidentify steps corresponding to ‘take a start posture 410’, ‘sit down420’, ‘stand up 430’, and ‘stand upright 440’ constituting a squataction.

The processor 140 according to an embodiment may obtain referenceinformation for each step 402. Here, the reference information for eachstep 402 according to an embodiment may be stored in the memory 130included in the electronic apparatus 100.

Specifically, reference information corresponding to ‘take a startposture 410’ may be threshold value information 411 corresponding to adifference between a leg width and a shoulder width. In addition,reference information corresponding to ‘sit down 420’ may be informationregarding an angle 421 between the thigh and shin and a waist curvature422. Reference information corresponding to ‘stand up 430’ may beinformation regarding a balance of the left and right sides of the body431 and a direction of gaze 432, and reference information correspondingto ‘stand upright 440’ may be information regarding a match rate withthe start posture.

The ideal form corresponding to the ‘sit down 420’ step according to anembodiment may be a posture in which the angle between the thigh andshin is 90 degrees, and the curvature of waist is equal to or greaterthan 1(1/m).

FIG. 5 is a view provided to explain a method of identifying a user'sexecution status based on posture information according to anembodiment.

The processor 140 according to an embodiment may identify the user'sexecution status for each step based on the identified referenceinformation for each step 402 and the user's context information. InFIG. 5 , it is assumed that the user's context information is theposture information of a user 301 who executes a squat action.

The processor 140 according to an embodiment may obtain postureinformation of the user 300 through skeleton analysis. The skeletonanalysis is an analysis method for analyzing actions included in animage.

In order for the skeleton analysis, the camera 120 provided in theelectronic apparatus 100 according to an embodiment may be implementedas a depth camera. Specifically, the processor 140 according to anembodiment may identify a point corresponding to a joint in the user'sbody included in an image obtained through a depth camera. In addition,the processor 140 may generate a skeleton model by connecting each pointbased on a positional relationship between points corresponding tojoints. The skeleton analysis is a prior art in the field related tomotion analysis and thus, further detailed descriptions will be omitted.

The processor 140 according to an embodiment may obtain a skeleton model310 of the user through skeleton analysis regarding an image of the user301 obtained through the camera 120. The processor 140 according to anembodiment may obtain the user's posture information based on the user'sskeleton model 310.

The processor 140 according to an embodiment may identify the user'sexecution status for each step based on the user's posture informationand reference information for each step 402.

Referring to FIG. 5 , a user is executing an action corresponding to‘sit down 420’ from among steps included in a content. The processor 140according to an embodiment may identify the user's execution statusbased on the posture information of the user 301 who executes the ‘sitdown 420’ action and reference information corresponding to the ‘sitdown 420’ action.

Specifically, the processor 140 may identify the user's execution statuscorresponding to the ‘sit down 420’ based on whether the user hasmaintained the angle between the thigh and shin at a certain angle andwhether the waist curvature is equal to or greater than a thresholdcurvature.

The processor 140 according to an embodiment may classify and identifythe execution status according to various criteria. Specifically, theprocessor 140 according to an embodiment may identify the user'sexecution status 500 based on the user's execution score 510, error rate520 and complexity of an action 530.

The execution score 510 according to an embodiment may be a numericalvalue identified based on how close the user's posture is to an idealform regarding the corresponding step. The processor 140 according to anembodiment may identify that the user's execution score 510 is 80 pointsbased on the degree of match between the user's posture and the idealform corresponding to the ‘sit down 420’ step.

When the user repeats an action corresponding to the ‘sit down 420’ stepmultiple times, the processor 140 according to an embodiment mayidentify the average of the execution scores corresponding to each roundas the final execution score 510 of the user.

The error rate 520 according to an embodiment means a probability ratethat the user executes an action in which the match rate with an idealform corresponding to a specific step is equal to or less than athreshold value. Specifically, when the user repeats an actioncorresponding to the ‘sit down 420’ step multiple times, the processor140 according to an embodiment may identify that the rate at which theuser executes an action indicating the match rate with an ideal formcorresponding to the ‘sit down 420’ step being 70 percent or less is 8percent.

The complexity 530 according to an embodiment means a level ofcomplexity for the user to follow an ideal form for a step.Specifically, the level of complexity corresponding to the ‘sit down420’ step may be ‘medium.’

When the above-described error rate 520 is determined based oncomplexity corresponding to each step, the processor 140 may determine athreshold value which is a criterion. Specifically, when the level ofcomplexity corresponding to a specific step is ‘easy’, the processor 140may determine that the threshold value which is a criterion fordetermining the error rate 520 is 90 percent. Meanwhile, when the levelof complexity is ‘medium’, the processor 140 may determine the thresholdvalue as 80 percent, and when the level of complexity is ‘hard’, theprocessor 140 may determine the threshold value as 70 percent.

FIGS. 6A and 6B are views provided to explain an operation ofidentifying an execution level corresponding to a user's executionstatus by an electronic apparatus according to various criteria andvarious types of feedbacks provided by the electronic apparatus based onexecution level the according to an embodiment.

Referring to FIG. 6A, the electronic apparatus 100 according to anembodiment may identify a user's execution level with reference to theexecution score 510, the error rate 520 and the complexity 530.

The processor 140 according to an embodiment may identify an executionlevel corresponding to an execution status which is classified based onthe numerical values of the execution score 510 and the error rate 520.Specifically, when the user's execution score exceeds 80 points (511),the processor 140 may identify the execution level corresponding to theuser's execution status classified as the execution score 510 to be ‘1’.

Meanwhile, when the user's execution score is greater than 50 and equalto or less than 80 (512), the processor 140 may identify the executionlevel corresponding to the user's execution status classified as theexecution score 510 to be ‘2’.

When the error rate is greater than 50 percent and equal to or less than80 percent (523), the processor 140 according to an embodiment mayidentify the execution level corresponding to the user's executionstatus classified as the error rate 520 to be ‘3’.

Meanwhile, when the error rate exceeds 80 percent (524), the processor140 according to an embodiment may identify the execution levelcorresponding to the user's execution status classified as the errorrate 520 to be ‘4’.

When the level of complexity is easy (531), the processor 140 accordingto an embodiment may identify the user's execution status classified asthe complexity 530 to be ‘1’. Since the case of the ‘hard’ complexity(533) requires more active interaction than the case of the ‘easy’complexity (531), the processor 140 according to an embodiment mayidentify the execution level corresponding to the user's executionstatus classified as the complexity 530 to be high as the complexityincreases.

Meanwhile, the electronic apparatus 100 according to an embodiment mayclassify execution statuses according to various criteria and then,provide various types of interactions 600 based on a plurality ofexecution levels obtained based thereon.

Specifically, interaction types according to an embodiment may includebasic control 610, speed control 620, focusing feedback 630, visualfeedback 640, posture correction 650, or mimic training 660.

The basic control 610 according to an embodiment may be an interactiontype corresponding to still image output or repeated output of subinformation corresponding to a specific step. Meanwhile, the visualfeedback 640 according to an embodiment may be an interaction type inwhich the electronic apparatus 100 outputs an image capturing the user'sposture through the camera 120 through the display 110, and theremaining interaction types will be described through the drawingslater.

In FIGS. 6A and 6B, a plurality of interaction types described above arerepresented by symbols such as ‘In (n is a natural number).’ From I1 toI6, it means that the electronic apparatus 100 can provide activeinteraction to the user.

The processor 140 according to an embodiment may determine aninteraction type to be provided based on the user's posture 540 as wellas the execution level. Specifically, the processor 140 may determinewhether to provide an interaction for posture correction 650 based onwhether the user can gaze at the display and the level of concentrationrequired to maintain the current posture of the user. This is because ifthe user cannot gaze at the display, there is a great need for posturecorrection using a separate driving unit (not shown) provided in theelectronic apparatus 100, and if a lot of concentration is required forthe user to maintain the current posture, performing posture correctionin a physical method is more advantageous than controlling the outputtype of a content through the display.

In addition, when the electronic apparatus 100 is implemented as arobot, the processor 140 may determine whether to provide a mimictraining 660 interaction based on a specification 550 of the robot.Specifically, when a robot according to an embodiment is implemented ina humanoid type, the robot may directly execute an ideal formcorresponding to sub information provided through a content in order toguide a posture that the user intends to take for a specific step.

FIG. 6B illustrates various types of feedbacks provided by theelectronic apparatus as a table based on an execution level. S on thevertical axis is a symbol representing an execution score, and E and Con the horizontal axis are symbols representing an error rate andcomplexity, respectively.

Specifically, when the sum of the execution level corresponding to theexecution status classified as the error rate 520 and the executionlevel corresponding to the execution status classified as the complexity530 is 3 and the execution level corresponding to the execution statusclassified as the execution score is 1 (601), the electronic apparatusaccording to an embodiment may provide an interaction of the basiccontrol 610.

Meanwhile, when the sum of the execution level corresponding to theexecution status classified as the error rate 520 and the executionlevel corresponding to the execution status classified as the complexity530 is 4 and the execution level corresponding to the execution statusclassified as the execution score is 2 (602), the electronic apparatusaccording to an embodiment may provide an interaction of the speedcontrol 620.

Meanwhile, when the sum of the execution level corresponding to theexecution status classified as the error rate 520 and the executionlevel corresponding to the execution status classified as the complexity530 is 6 and the execution level corresponding to the execution statusclassified as the execution score is 2 (603), the electronic apparatusaccording to an embodiment may provide an interaction of the focusingfeedback 630.

Meanwhile, when the sum of the execution level corresponding to theexecution status classified as the error rate 520 and the executionlevel corresponding to the execution status classified as the complexity530 is 5 and the execution level corresponding to the execution statusclassified as the execution score is 4 (604), the electronic apparatusaccording to an embodiment may provide an interaction of the visualfeedback 640.

Meanwhile, when the sum of the execution level corresponding to theexecution status classified as the error rate 520 and the executionlevel corresponding to the execution status classified as the complexity530 is 7 and the execution level corresponding to the execution statusclassified as the execution score is 4 (605), the electronic apparatusaccording to an embodiment may provide an interaction of the visualfeedback 640. In this case, the processor 140 may provide an interactionof the posture correction 650 or the mimic training 660.

FIGS. 7A and 7B are views provided to explain a posture correctionoperation and a mimic training operation of an electronic apparatusaccording to an embodiment.

Referring to FIG. 7A, an electronic apparatus 101 according to anembodiment may include a driving unit 710. Unlike the description forFIG. 2 , FIG. 7A illustrates that the driving unit 710 includesmechanical parts provided in the electronic apparatus 101, and not onlythe parts in the form of a robot shown therein but also the parts in theform of a wheel provided at the bottom of the electronic apparatus 100and the parts in the form of an arm coupled to the real surface of thedisplay 110 may also be included in the driving unit 710.

When it is identified that the user 301 requires a posture correctiontype of interaction, the electronic apparatus 101 may correct a wrongposture of the user 301 through the driving unit 710. The electronicapparatus 100 may touch the user 301 through the driving unit 710 in theprocess of providing the corresponding interaction.

In addition, the electronic apparatus 100 may move the position of thedisplay 100 closer to the user 301 or adjust the angle of the display100.

Referring to FIG. 7B, an electronic apparatus 102 according to anembodiment may be implemented as a humanoid type robot. Although notillustrated in FIG. 7B, the humanoid type robot 102 may include adriving unit including integrated configuration for driving the robot.

The robot 102 according to an embodiment may provide a mimic trainingtype of interaction to a user by following an action corresponding to aspecific step included in a content provided through the display 110.

In addition, the robot 102 according to an embodiment may include aspeaker 720, and control the speaker 720 to output a voice of “pleasefollow me” while providing a mimic training type of interaction to theuser 301.

FIGS. 8A to 8C are views provided to explain an operation of controllingan output state of a content by recognizing a user's facial expressionby an electronic apparatus according to an embodiment.

Referring to FIG. 8A, the electronic apparatus 100 according to anembodiment may provide an origami-related content that provides taskinformation including a plurality of steps to a user 302. The processor140 according to an embodiment may identify a plurality of stepsincluded in an origami action.

The processor 140 according to an embodiment may identify ‘a result’corresponding to each step as an ideal form for each step, and identifythe execution status of the user 302 by comparing the result made by theuser 302 and the ideal form.

The processor 140 according to an embodiment may provide an interactionregarding the user 302 based on the identified execution status. Whenthe user does not make a result corresponding to a step, the processor140 may repeatedly output sub information corresponding to thecorresponding step. Meanwhile, the processor 140 may output subinformation corresponding to the first step included in a content andcontinue to output sub information corresponding to subsequent steps.

In addition, when supplies 801, 802, 803 are not prepared, the processor140 may provide a predetermined interaction prior to identifying theuser's execution status.

The processor 140 according to an embodiment may identify the face ofthe user 302 based on an image obtained through the camera 120. Theprocessor according to an embodiment may obtain the user's facialexpression information based on the face image of the user.

The processor 140 according to an embodiment may obtain the user'sfacial expression information by inputting the face image to the firstneural network model, and the first neural network model may be a modeltrained to receive a plurality of images and output an image including afrowning face.

The processor 140 according to an embodiment may identify that a usermakes a frowning expression (810), and control the display 110 toprovide guide information corresponding to a step requiring a feedback.Specifically, the processor 140 may control the display 110 to displayguide information in which the output type of sub informationcorresponding to a step requiring a feedback has changed.

Specifically, guide information according to an embodiment may identifya step corresponding to sub information that is being provided at themoment when the user 302 makes a frowning expression from among stepsincluded in an origami action, and provide a UI informing that subinformation is provided again while providing sub information of thecorresponding step again.

In addition, referring to FIG. 8B, the electronic apparatus 100according to an embodiment may identify that a user makes a frowningexpression (810), and immediately stop providing a content (821).Meanwhile, the processor 140 as shown in FIG. 8C may identify that auser makes a frowning expression (810), and immediately adjust theoutput speed of the content (822).

FIGS. 9A to 9C are views provided to explain an operation of controllingan output state of a content by recognizing a user's voice by anelectronic apparatus according to an embodiment.

Referring to FIG. 9A, the electronic apparatus 100 according to anembodiment may provide a cooking-related content that provides taskinformation including a plurality of steps to a user 303. The processor140 according to an embodiment may identify a plurality of stepsincluded in a cooking action.

The processor 140 according to an embodiment may identify ‘a result’corresponding to each step as an ideal form for the step, and identifythe execution status of the user 302 by comparing the result made by theuser 303 with the ideal form.

The processor 140 according to an embodiment may provide an interactionregarding the user 303 based on the identified execution status. Whenthe user does not make a result corresponding to the step, the processor140 may repeatedly output sub information corresponding to thecorresponding step. Meanwhile, the processor 140 may output subinformation corresponding to the first step included in the content andcontinue to output sub information corresponding to subsequent steps.

In addition, when supplies 901, 902, 903, 904 are not prepared, theprocessor 140 may provide a predetermined interaction prior toidentifying the user's execution status.

The processor 140 according to an embodiment may recognize the user'sutterance. Specifically, the processor 140 may recognize the utterancebased on the user's voice information obtained through a microphone (notillustrated) that is provided separately or recognize the utterancebased on the user's mouth shape image obtained through the camera 120.

When the processor 140 according to an embodiment recognizes anutterance based on voice information, the processor 140 may obtain theuser's utterance information by inputting the voice information to thesecond neural network model, and the second neural network model may bea model trained to receive a plurality of pieces of voice informationand output a voice including an utterance describing an unpleasant mood.

The processor 140 according to an embodiment may identify an utterancedescribing the user's unpleasant mood and accordingly, control thedisplay 110 to provide guide information corresponding to a steprequiring a feedback. Specifically, the processor 140 may control thedisplay 110 to display guide information in which the output type of subinformation corresponding to a step requiring a feedback has changed.

Specifically, the guide information according to an embodiment mayidentify a step corresponding to sub information that is being providedat the moment when the user 303 utters from among steps included in acooking action, and provide a UI informing that sub information isprovided again while providing sub information of the corresponding stepagain.

In addition, referring to FIG. 9B, the electronic apparatus 100according to an embodiment may identify the user's utterance 910, anddisplay a UI 921 requesting an input regarding whether to stop providinga content. Meanwhile, as illustrated in FIG. 9C, the processor 140 mayidentify the user's utterance 910, and enlarge a specific part 922 ofthe screen provided through the content. This may be an interaction typecorresponding to the focusing feedback 630 described in FIG. 6A.

Meanwhile, when it is identified that the user 303 is having aconversation with another person based on the user's voice receivedthrough a microphone, the processor 140 according to an embodiment maycontrol the display 110 to stop providing the content.

As described in FIGS. 9A and 9B, the electronic apparatus 100 mayprovide various interactions based on the user's level of understandingregarding information included in the content.

FIG. 10 is a block diagram provided to specifically explain functionalconfiguration of an electronic apparatus according to an embodiment.

Referring to FIG. 10 , an electronic apparatus 100′ includes the display110, the camera 120, the memory 130, the processor 140, a speaker 150, acommunication interface 160, a driving unit 170, and a microphone 180.Among components illustrated in FIG. 10 , detailed descriptionsregarding the components which overlap with those illustrated in FIG. 2will be omitted.

The memory 130 according to an embodiment may store informationregarding a first neural network model 131 and a second neural networkmodel 132 including a plurality of layers. Here, storing informationregarding a neural network model may mean storing various informationrelated to the operations of a neural network model, for example,information regarding a plurality of layers included in the neuralnetwork model, information regarding a parameter (e.g., filtercoefficients, biases, etc.) used in each of the plurality of layers,etc. For example, the memory 130 may store information regarding thefirst neural network model 131 trained to receive a plurality of imagesand output an image including a frowning facial expression and the storeinformation regarding the second neural network model 132 trained toreceive a plurality of pieces of voice information and output a voiceincluding an utterance describing an unpleasant mood according to anembodiment.

The speaker 150 is a device that converts an electrical acoustic signalof the electronic apparatus 100 into sound waves. The speaker 150 mayinclude a permanent magnet, a coil and a diaphragm, and may output soundby vibrating the diaphragm by electromagnetic interaction that occursbetween the permanent magnet and the coil.

The communication interface 160 may input and output various types ofdata. For example, the communication interface 160 may transmit/receivevarious types of data to/from an external device (e.g., source device),an external storage medium (e.g., universal serial bus (USB) memory), anexternal server (e.g., web hard) or the like by using a communicationmethod such as an access point (AP)-based wireless fidelity (Wi-Fi, i.e.wireless local area network (LAN)), a Bluetooth, a Zigbee, awired/wireless local area network (LAN), a wide area network (WAN),Ethernet, an IEEE 1394, a high definition multimedia interface (HDMI), aUSB, a mobile high-definition link (MHL), an audio engineeringsociety/European broadcasting union (AES/EBU) communication, an opticalcommunication or a coaxial communication.

When the electronic apparatus 100 provides a user with an interaction,the driving unit 170 may drive the electronic apparatus 100 under thecontrol of the processor 140. Specifically, the driving unit 170 maymove the position of the electronic apparatus 100 or drive mechanicalparts included in the electronic apparatus 100. To this end, the drivingunit 170 may include a power generating device that generates power(e.g., a gasoline engine, a diesel engine, a liquefied petroleum gasengine, an electric motor, etc. depending on the fuel (or energy source)used), a steering device (e.g., manual steering, hydraulics steering,electronic control power steering (EPS), etc.) for adjusting a drivingdirection, a driving device (e.g., wheels, propellers, etc.) for drivingthe electronic apparatus 100 according to power. Here, the driving unit170 may be provided in different forms based on the type in which theelectronic apparatus 100 is implemented.

The microphone 180 is configured to receive a sound signal.Specifically, the microphone 180 is configuration that collectivelyrefers to a device that receives a sound wave and generates a current ofthe same waveform. In the above drawings, it is described that a signalincluding a user's voice is received, but the microphone 180 accordingto an embodiment may receive various sound signals such as footsteps,breathing sounds, plosive sounds, etc.

Specifically, the processor 140 may identify whether a user who isexecuting a running action is exercising at an appropriate speed basedon an interval of generation of footsteps received through themicrophone 180. The processor 140 may identify whether the user'sbreathing is stable based on the breathing sound received through themicrophone 180.

In addition, the processor 140 may identify that the user has beeninjured due to a cause such as a fall during exercise based on theplosive sound received through the microphone 180 and then, provide acorresponding interaction. Accordingly, the electronic apparatus 100 mayrecognize a situation more accurately when reliability of imageinformation obtained through the camera 120 is low.

FIG. 11 is a flowchart provided to explain a controlling methodaccording to an embodiment.

A controlling method of an electronic apparatus according to anembodiment may include obtaining an image capturing a user through acamera while a content that provides information is being displayed(S1110). Subsequently, context information related to the user isobtained based on the image (S1120). By comparing the obtained contextinformation and reference information related to the content, the userexecution status regarding the information provided by the content isidentified (S1130). Lastly, the output state of the content iscontrolled based on the user's execution status (S1140).

Here, the content includes sub information corresponding to each of aplurality of steps, and the identifying the user's execution status(S1130) may include identifying the user's execution status regardingone step by comparing the obtained user's context information andreference information corresponding to sub information while the subinformation corresponding to one step from among a plurality of steps isprovided through the display, and the controlling the output state ofthe content (S1140) may include controlling the output state of thecontent related to one step based on the user's execution status.

Here, the identifying the user's execution status (S1130) may includeidentifying an execution level corresponding to the user's executionstatus, and the controlling the output state of the content (S1140) mayinclude identifying information regarding the output type of the contentcorresponding to the identified execution level and controlling theoutput state of the content based on the identified output type.

Here, the output type of the content may include at least one of stillimage output, repeated output of sub information corresponding to aspecific step, output speed adjustment, or enlarged output of thecontent.

Meanwhile, the obtaining context information related to the user (S1120)may include obtaining the user's posture information based on an imageobtained while a content that provides health information including anexercise posture is provided through the display, and the identifyingthe user's execution status (S1130) may include identifying the user'sexecution status regarding the exercise posture by comparing theobtained posture information and posture information included inreference information and providing a different feedback for guiding theuser's posture based on the user's execution status.

Here, the electronic apparatus 100 is implemented as a robot thatprovides a healthcare service, and the providing a different feedbackmay include performing an operation of correcting the user's posture orguiding the user's posture so that the user can execute the exerciseposture provided by the content.

In addition, the identifying the user's execution status (S1130) mayfurther include identifying the user's execution level based on at leastone of a frequency of discrepancy between the obtained postureinformation and posture information included in the referenceinformation or a complexity of the exercise posture, and the providing adifferent feedback may include providing a feedback corresponding to theidentified execution level.

Meanwhile, the obtaining context information related to the user (S1120)may include obtaining the user's status information based on the imageobtained while the content that provides task information including aplurality of steps is provided through the display, and the identifyingthe user's execution status (S1130) may include identifying the user'sexecution status for each of the plurality of steps by comparing theobtained status information and status information included in referenceinformation and providing guide information corresponding to a steprequiring a feedback from among the plurality of steps based on theuser's execution status.

Here, the obtaining the user's status information may include obtainingthe user's first status information by inputting the image to the firstneural network model and obtaining the user's second status informationby inputting the image to the first neural network model and obtainingthe user's second status information by inputting the user's voice tothe second neural network model, and the providing guide information mayinclude providing guide information corresponding to a step requiring afeedback from among a plurality of steps based on the first statusinformation and the second status information.

In addition, when the user's image is not obtained while informationregarding one of the plurality of steps is provided, guide informationrelated to task information may be output.

Meanwhile, the methods according to various embodiments of thedisclosure described above may be implemented in the form of anapplication that may be installed in an existing electronic apparatus.

In addition, the methods according to the diverse embodiments of thedisclosure described above may be implemented only by software upgradeor hardware upgrade for the existing electronic apparatus.

In addition, various embodiments of the disclosure described above maybe implemented by an embedded server included in an electronic apparatusor at least one external server.

Meanwhile, the various embodiments described above may be implemented ina computer or a computer-readable recording medium using software,hardware, or a combination of software and hardware. In some cases, theembodiments described in the present disclosure may be implemented bythe processor 140 itself. According to a software implementation, theembodiments such as the procedures and functions described in thepresent disclosure may be implemented by separate software modules. Eachof the software modules may perform one or more functions and operationsdescribed in the specification.

Meanwhile, a non-transitory computer-readable medium may store computerinstructions for performing the processing operations of the electronicapparatus 100 according to the various embodiments of the presentdisclosure described above. The computer instructions stored in thenon-transitory computer-readable medium may allow a specific device toperform the processing operations of the electronic apparatus 100according to the various embodiments described above when the computerinstructions are executed by a processor of the specific device.

The non-transitory computer-readable medium is not a medium that storesdata therein for a while, such as a register, a cache, or a memory, andindicates a medium that semi-permanently stores data therein and isreadable by the machine. A specific example of the non-transitorycomputer-readable medium may include a compact disk (CD), a digitalversatile disk (DVD), a hard disk, a Blue-ray disk, a universal serialbus (USB), a memory card, a read-only memory (ROM), or the like.

Although the embodiments are shown and described in the presentdisclosure as above, the present disclosure is not limited to theabove-mentioned specific embodiments, and may be variously modified bythose skilled in the art to which the present disclosure pertainswithout departing from the gist of the present disclosure as claimed inthe accompanying claims. These modifications should also be understoodto fall within the scope and spirit of the present disclosure.

Although embodiments of the disclosure have been illustrated anddescribed herein, the disclosure is not limited thereto, and variousmodifications may be made by those of ordinary skill in the art withoutdeparting from the gist of the disclosure defined in the appended claimsand should not be understood separately from the technical idea orprospect of the disclosure.

What is claimed is:
 1. An electronic apparatus comprising: a display; acamera; a memory in which reference information associated with acontent that provides information is stored; and a processor configuredto: obtain an image photographing a user through the camera while thecontent is provided through the display; obtain context informationassociated with the user based on the image; identify an executionstatus of the user associated with the information provided by thecontent by comparing the obtained context information and the referenceinformation; and control an output state of the content based on theexecution status of the user.
 2. The electronic apparatus as claimed inclaim 1, wherein the content includes respective sub informationcorresponding to each of a plurality of steps; and wherein the processoris configured to; while the respective sub information corresponding toa first step from among the plurality of steps is provided through thedisplay, identify the execution status of the user associated with thefirst step by comparing the obtained context information and referenceinformation corresponding to the respective sub information; and controlthe output state of the content corresponding to the first step based onthe execution status of the user.
 3. The electronic apparatus as claimedin claim 2, wherein the memory stores a type of an output content foreach execution level of the user; and wherein the processor isconfigured to: identify an execution level corresponding to theexecution status of the user; identify information associated with thetype of the output content corresponding to the identified executionlevel; and control the output state of the content based on the type ofthe output content.
 4. The electronic apparatus as claimed in claim 3,wherein the type of the output content includes at least one of a stillimage output, repeated output of the respective sub informationcorresponding to a specific step, output speed adjustment, or enlargedoutput of contents.
 5. The electronic apparatus as claimed in claim 1,further comprising: a driving unit, wherein the processor is configuredto: obtain posture information of the user based on the image obtainedwhile a content that provides health information including an exerciseposture is provided through the display; identify the execution statusof the user associated with the exercise posture by comparing theobtained posture information of the user and the posture informationincluded in the reference information; and control the driving unit toprovide a specific feedback for guiding the user's posture based on theexecution status of the user.
 6. The electronic apparatus as claimed inclaim 5, wherein the electronic apparatus is implemented as a robot thatprovides a healthcare service; and wherein the robot performs anoperation of correcting the user's posture or provides guide informationfor guiding the user's posture so that the user executes the exerciseposture provided by the content based on the execution status of theuser.
 7. The electronic apparatus as claimed in claim 5, wherein theprocessor is configured to identify an execution level of the user basedon at least one of a degree of correspondence between the obtainedposture information and the posture information included in thereference information, a frequency of discrepancy between the obtainedposture information and the posture information included in thereference information, or complexity of the exercise posture, andprovide a feedback corresponding to the identified execution level. 8.The electronic apparatus as claimed in claim 1, wherein the processor isconfigured to: obtain a status information of the user based on theimage obtained while the content is provided through the display, thecontent including information corresponding to a plurality of steps;identify the execution status of the user associated with each of theplurality of steps by comparing the obtained status information and thestatus information included in the reference information; and controlthe display to provide guide information corresponding to a steprequiring feedback based on the execution status of the user, whereinthe step is from among the plurality of steps.
 9. The electronicapparatus as claimed in claim 8, wherein the processor is configured to:obtain a first status information of the user by inputting the image toa first neural network model; obtain a second status information of theuser by inputting a voice of the user to a second neural network model;and control the display to provide the guide information correspondingto a step requiring a feedback based on the identified execution status,the first status information and the second status information, whereinthe step is from among the plurality of steps.
 10. The electronicapparatus as claimed in claim 8, further comprising: a speaker, whereinthe processor is configured to, based on the image of the user not beingobtained while information associated with one step from among theplurality of steps is provided, control the speaker to output the guideinformation associated with the content.
 11. A controlling method of anelectronic apparatus, comprising: obtaining an image photographing auser through a camera while a content that provides information isdisplayed; obtaining context information associated with the user basedon the image; identifying an execution status of the user associatedwith the information provided by the content by comparing the obtainedcontext information and the reference information associated with thecontent; and controlling an output state of the content based on theexecution status of the user.
 12. The method as claimed in claim 11,wherein the content includes respective sub information corresponding toeach of a plurality of steps; wherein the identifying the executionstatus of the user comprises, while the respective sub informationcorresponding to a first step from among the plurality of steps isprovided through a display, identifying the execution status of the userassociated with the first step by comparing the obtained contextinformation and reference information corresponding to the respectivesub information; and wherein the controlling the output state of thecontent comprises controlling the output state of the content associatedwith the first step based on the execution status of the user.
 13. Themethod as claimed in claim 12, wherein the identifying the executionstatus of the user comprises identifying an execution levelcorresponding to the execution status of the user; and wherein thecontrolling the output state of the content comprises identifyinginformation associated with a type of an output content corresponding tothe identified execution level and controlling the output state of thecontent based on the type of the output content.
 14. The method asclaimed in claim 13, wherein the type of the output content includes atleast one of a still image output, repeated output of sub informationcorresponding to a specific step, output speed adjustment, or enlargedoutput of contents.
 15. The method as claimed in claim 11, wherein theobtaining context information associated with the user comprisesobtaining posture information of the user based on the image obtainedwhile the content that provides health information including an exerciseposture is provided through a display; and wherein the identifying theexecution status of the user comprises: identifying the execution statusof the user associated with the exercise posture by comparing theobtained posture information of the user and posture informationincluded in the reference information; and providing a specific feedbackfor guiding the user's posture based on the execution status of theuser.
 16. The method as claimed in claim 15, wherein the electronicapparatus is implemented as a robot that provides a healthcare service;and wherein the providing the specific feedback comprises: performing anoperation of correcting the user's posture or providing guideinformation for guiding the user's posture so that the user executes theexercise posture provided by the content based on the execution statusof the user.
 17. The electronic apparatus as claimed in claim 5, whereinthe identifying the execution status of the user comprises: identifyingthe execution level of the user based on at least one of a degree ofcorrespondence between the obtained posture information and the postureinformation included in the reference information, a frequency ofdiscrepancy between the obtained posture information and the postureinformation included in the reference information, or complexity of theexercise posture, and wherein the providing the specific feedbackcomprises: providing a feedback corresponding to the identifiedexecution level.
 18. The method as claimed in claim 11, wherein theobtaining context information associated with the user comprises:obtaining a status information of the user based on the image obtainedwhile the content is provided through a display, the content includinginformation corresponding to a plurality of steps; wherein theidentifying the execution status of the user comprises: identifying theexecution status of the user associated with each of the plurality ofsteps by comparing the obtained status information and the statusinformation included in the reference information; and providing guideinformation corresponding to a step requiring feedback based on theexecution status of the user, wherein the step is from among theplurality of steps.
 19. The method as claimed in claim 18, wherein theobtaining the status information of the user comprises: obtaining afirst status information of the user by inputting the image to a firstneural network model; obtaining a second status information of the userby inputting a voice of the user to a second neural network model; andwherein the providing the guide information comprises: providing theguide information corresponding to a step requiring a feedback based onthe identified execution status, the first status information and thesecond status information, wherein the step is from among the pluralityof steps.
 20. A non-transitory computer readable recording mediumstoring computer instructions that cause an electronic apparatus toperform an operation when executed by a processor of the electronicapparatus, wherein the operation comprises; obtaining an imagephotographing a user through a camera while a content that providesinformation is displayed; obtaining context information associated withthe user based on the image; identifying an execution status of the userassociated with the information provided by the content by comparing theobtained context information and the reference information associatedwith the content; and controlling an output state of the content basedon the execution status of the user.